Productionize claims_hosp #1408

korlaxxalrok · 2021-12-09T20:13:44Z

Description

Adds the necessary Ansible bits and scripts to deploy claims_hosp like an indicator while keeping intact its current automation routines.

Changelog

Adds additonal directory structure and current automation scripts.
Gives Ansible the ability to place a templated secrets.py file into an indicator's root directory.
Updates vars and vault with new secrets.

Start from Jingjing

All the pipeline required code are moved to delphi_claims_hosp.
HospClaims could be dropped. But didn't do that currently just for a backup.
Switch from print() to logger
Add pulling steps from shell scripts to run.py
Pulled files are downloaded and stored in ./retrieve_files. All the raw files will be deleted after a production run
refactor secrets into params.json
Check the validation results. The type of hrr geo_ids are preferred to be string. However, if I change it to zero-filled strings, it does not match the history which generates more problems. So, switched back.

Fixes

Fixes #1623

- Updates deployment playbooks for prod and staging to handle the possibility of a `secrets.py` file. - Updates vars and vault files with new variable pointers and encryption. - Adds a new template file that will be rendered as `secrets.py` in the root of the indicators directory.

- Adds new Python and shell scripts to automate `hosp_claims` indicator. - Adds a symlink for `secrets.py`.

ansible/ansible-deploy-staging.yaml

krivard

Looks like this still has a ways to go before it's ready to deploy

claims_hosp/HospClaims/automate/agg_claims_drops.py

claims_hosp/HospClaims/automate/README.md

claims_hosp/HospClaims/automate/agg_claims_drops.py

claims_hosp/HospClaims/automate/download_claims_ftp_files.py

claims_hosp/HospClaims/automate/sanity_checks.py

claims_hosp/HospClaims/automate/secrets.py

claims_hosp/HospClaims/automate/update_json.py

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

korlaxxalrok · 2021-12-10T13:56:35Z

@mariajahja Can you take a quick pass through this, maybe check out those comment-cleanup questions?

claims_hosp/HospClaims/automate/agg_claims_drops.py

claims_hosp/HospClaims/automate/hosp_claims_master_script.sh

mariajahja · 2021-12-11T20:04:39Z

claims_hosp/HospClaims/automate/hosp_claims_master_script.sh

+sanity_check() {
+  geo=$1
+  cd "$AUTO_DIR" || exit
+  python3 sanity_checks.py "$RECEIVING_DIR" "$geo"


This script performs very basic sanity checks (e.g. percentages between 0 and 100) and can also produce some plots contrasting the new values to ones in the API (see a few line below). If there is a separate validation script run on other indicators such as claims, it would be better to use that and remove this.

claims_hosp/HospClaims/automate/hosp_claims_master_script.sh

claims_hosp/HospClaims/automate/hosp_claims_regen_script.sh

claims_hosp/HospClaims/automate/regen_old_issue.py

claims_hosp/HospClaims/automate/ftp_to_covidcast.py

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

…idcast-indicators into productionize-claims_hosp

jingjtang · 2022-06-15T14:28:39Z

@korlaxxalrok @krivard Could you help me with resolving the conflicts in ansible/vault.yaml? What should I do with it?

krivard · 2022-06-15T17:01:30Z

I'll fix the vault; we had an update to the safegraph secrets while this PR was in flight

qx-teo

Looks good on the validator end!

claims_hosp/delphi_claims_hosp/download_claims_ftp_files.py

krivard

question on need for force parameter
additional information needed in final log statement

claims_hosp/delphi_claims_hosp/modify_claims_drops.py

claims_hosp/delphi_claims_hosp/run.py

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

krivard

Way closer! Two minor items to resolve.

claims_hosp/delphi_claims_hosp/modify_claims_drops.py

krivard · 2022-06-21T18:14:47Z

claims_hosp/.pylintrc

    # Allow pytest classes to have one test.
-    too-few-public-methods
+    too-few-public-methods,
+    broad-except


@jingjtang why is broad-except needed?

jingjtang · 2022-06-21T23:32:33Z

why is broad-except needed?

@krivard It's for this warning:

delphi_claims_hosp/download_claims_ftp_files.py:36:11: W0703: Catching too general exception Exception (broad-except)

There can be multiples types of exception error (IndexError, ValueError, ..) if we run into unexpected names

Remove try/except from get_timestamp by converting to re

krivard

👍

@korlaxxalrok is any preparation needed before we merge this or can I click the big green button at will?

korlaxxalrok · 2022-06-30T17:04:18Z

@krivard Looks reasonable. As best I can tell we won't need any prep, but imo best path is to merge and test 👍

korlaxxalrok added 5 commits December 8, 2021 15:57

Add hosp_claims automation rig

c3227c4

- Adds new Python and shell scripts to automate `hosp_claims` indicator. - Adds a symlink for `secrets.py`.

Merge branch 'main' into productionize-claims_hosp

9d30a69

Merge branch 'main' into productionize-claims_hosp

17c35f7

Re-add new secrets after fixing merge conflict with main

095f792

korlaxxalrok requested review from krivard and mariajahja December 9, 2021 20:32

krivard reviewed Dec 9, 2021

View reviewed changes

ansible/ansible-deploy-staging.yaml Outdated Show resolved Hide resolved

Register template and template-secrets in case we have both

b70e0bb

krivard reviewed Dec 9, 2021

View reviewed changes

Update claims_hosp/HospClaims/automate/update_json.py

ab1c2b8

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

mariajahja reviewed Dec 11, 2021

View reviewed changes

claims_hosp/HospClaims/automate/ftp_to_covidcast.py Outdated Show resolved Hide resolved

korlaxxalrok and others added 5 commits May 3, 2022 09:39

Remove unused lines

4193038

Update claims_hosp/HospClaims/automate/update_json.py

17b85d5

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

In light of #1419 being merged, we can change to 70 dates

a631d77

Merge branch 'productionize-claims_hosp' of github.com:cmu-delphi/cov…

d6dff50

…idcast-indicators into productionize-claims_hosp

Remove unused line

126f6e1

korlaxxalrok marked this pull request as draft May 3, 2022 17:53

Jingjing Tang added 10 commits June 6, 2022 07:52

add scripts and modification required for pulling data

d9b087f

rm downloaded files at the end

0e59cd1

add logger info for it

c927e00

fixed errors in pulling

c69cc2f

add unit tests

40411f8

change the path to test.log

b753b7e

fixed an error

edf8e3f

change the format of geo_id at hrr level from float to str

2d4400b

fix hrr geo id type

a7f9146

add unit tests

a2fcfcb

Jingjing Tang added 4 commits June 15, 2022 10:19

update the function for renmaing the raw drops

0be164c

remove agg function

2c9b21d

update import info for the modification function

080ad45

add j2 file for claims_hosp params

819b043

jingjtang requested a review from krivard June 15, 2022 14:29

krivard added 2 commits June 15, 2022 14:47

Manually merge vault changes

cfb682c

Merge branch 'main' into productionize-claims_hosp

a8c2a6e

jingjtang requested a review from qx-teo June 16, 2022 04:47

qx-teo approved these changes Jun 16, 2022

View reviewed changes

krivard reviewed Jun 16, 2022

View reviewed changes

claims_hosp/delphi_claims_hosp/download_claims_ftp_files.py Outdated Show resolved Hide resolved

krivard suggested changes Jun 16, 2022

View reviewed changes

claims_hosp/delphi_claims_hosp/modify_claims_drops.py Outdated Show resolved Hide resolved

claims_hosp/delphi_claims_hosp/run.py Outdated Show resolved Hide resolved

jingjtang and others added 4 commits June 16, 2022 13:51

Update logger info

6f4976c

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

add min_max_dates, csv_export_count, etc to logger info

1c83ffd

update the unit test for modification of the raw files

50c93fb

fix the error in the unit test

300397f

jingjtang requested a review from krivard June 21, 2022 13:12

krivard suggested changes Jun 21, 2022

View reviewed changes

rename the parameter for test_mode check

76b4db0

jingjtang requested a review from krivard June 21, 2022 23:55

krivard added 3 commits June 22, 2022 16:32

Remove try/except from get_timestamp by converting to re

aef9511

Update expected file count for inpatient-only

170f2e3

Merge pull request #1644 from cmu-delphi/krivard/pch-no-general-except

d1ffb14

Remove try/except from get_timestamp by converting to re

krivard approved these changes Jun 27, 2022

View reviewed changes

krivard merged commit 74cb9e1 into main Jun 30, 2022

krivard deleted the productionize-claims_hosp branch June 30, 2022 17:28

krivard mentioned this pull request Jul 5, 2022

Release covidcast-indicators 0.3.17 #1653

Merged

Productionize claims_hosp #1408

Productionize claims_hosp #1408

Uh oh!

Conversation

korlaxxalrok commented Dec 9, 2021 • edited by jingjtang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog

Fixes

Uh oh!

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

korlaxxalrok commented Dec 10, 2021

Uh oh!

Uh oh!

Uh oh!

mariajahja Dec 11, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jingjtang commented Jun 15, 2022

Uh oh!

krivard commented Jun 15, 2022

Uh oh!

qx-teo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

krivard Jun 21, 2022

Choose a reason for hiding this comment

Uh oh!

jingjtang commented Jun 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

korlaxxalrok commented Jun 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

korlaxxalrok commented Dec 9, 2021 •

edited by jingjtang

Loading

jingjtang commented Jun 21, 2022 •

edited

Loading